docs(api): document exact /status schema separating process vs data health (#273) - #533

Merged as 794eb07. The framing is the valuable part here: "do not page on upstream ledger delay, do page on Postgres being gone" is a real operational distinction, and encoding it as a 200-with-degraded versus a 503-with-pagingCondition is the right way to make it machine-readable. Keeping the old top-level lastIndexedLedger / lastProcessedAt / ingestLagSeconds alongside the new data block means no existing consumer breaks, which is easy to forget on a monitored endpoint. The Datadog/Prometheus/BetterStack examples in the README are the kind of thing that actually gets copied.

Two fixes before it went in.

1. The 503 returned the Postgres driver's message on a public endpoint (blocking)

} catch (err: any) {
  return reply.status(503).send({
    ok: false,
    status: 'unhealthy',
    error: err?.message || 'Database query failed',
    pagingCondition: 'DATABASE_DOWN',
  })
}

/status is config: { public: true } — no API key, no admin token — and it is the most-polled surface Lens has (UptimeRobot hits it continuously). A pg connection failure does not produce a tidy message; it produces things like:

connect ECONNREFUSED: password authentication failed for user "lens_admin"
(postgresql://lens_admin:hunter2@db.internal.example:5432/lens)

Lens has already had to rotate a database password over this exact shape of leak on /price/twap, where a raw pg error was echoed to the client and sat there for roughly three months on a database shared with another service. So this is not a hypothetical — it is the same bug on a louder route.

What makes it worth flagging rather than just fixing quietly is that the test pinned the leak as the intended behaviour:

expect(body.error).toContain('Connection terminated')

A test asserting a leaked driver message is a test that will defend it against the next person who tries to remove it. I inverted it, and added a second case that feeds in a connection-string-bearing error and asserts none of hunter2, lens_admin, db.internal.example or postgresql:// appears anywhere in the serialized response.

The fix keeps the diagnostic, it just sends it to the right place:

} catch (err: unknown) {
  req.log.error({ err }, 'status: database query failed')
  return reply.status(503).send({
    ok: false,
    status: 'unhealthy',
    error: 'Database unavailable',
    pagingCondition: 'DATABASE_DOWN',
  })
}

buildLogger pipes log lines through redactSecrets (src/logger.ts), which knows about DATABASE_URL and the password inside it — so the log keeps the full driver text with the credentials scrubbed, which is exactly what an operator needs and the client does not. I also gave statusErrorResponseSchema additionalProperties: false, matching the 200 schema, so a future detail field cannot quietly reopen this.

Nothing is lost for the monitor: status_code == 503 || body.pagingCondition == 'DATABASE_DOWN' from your own README example still fires identically.

2. Process health was a constant

const processHealth = 'healthy' as const

health.process and process.status could never be anything but 'healthy', which means the degraded branch of the enum you published —

process:
  type: string
  enum: [healthy, degraded]

— was unreachable. So "separates process health from data health" was true on the data side and decorative on the process side: uptimeSeconds and memoryUsageMb are genuinely measured, but the health verdict derived from them was not.

It now reads a real signal:

const processHealth = isBootComplete() ? ('healthy' as const) : ('degraded' as const)

isBootComplete() (src/health.ts) is the same thing /ready gates on, and it is the one process-level state that is genuinely distinct from data freshness: between app.listen() and markBootComplete() the port answers requests while not every ingester is running yet. That is a degraded process with perfectly fresh data — the exact quadrant your two-axis model exists to express, and now the test for it asserts health.process: 'degraded' with health.data: 'healthy'.

One knock-on: statusContract.test.ts now calls markBootComplete() in beforeEach. A test that means "a healthy process" has to say so; without it buildApp() leaves the app mid-boot, which is the degraded case.

I also pulled the 300-second threshold out into a named constant. It is a published number — the README monitoring examples quote > 300 and the openapi degraded description depends on it — and a monitor tuned to one value while the service uses another is worse than no monitor at all.

Generated artifacts

openapi.json and clients/typescript/src/schema.d.ts are both guarded in CI, so both are regenerated in the merge commit (npm run openapi:gen, then openapi-typescript against openapi.yaml). Worth knowing for next time: npm run client:gen needs openapi-typescript on PATH, so locally it is npx --yes openapi-typescript@7.13.0 openapi.yaml -o clients/typescript/src/schema.d.ts.

125 test files / 1,486 tests pass. tsc --noEmit clean.